Learning Algorithms for Risk-Sensitive Control

This is a survey of some reinforcement learning algorithms for risk-sensitive control on infinite horizon. Basics of the risk-sensitive control problem are recalled, notably the corresponding dynamic programming equation and the value and policy iteration methods for its solution. Basics of stochastic approximation algorithms are also sketched, in particular the ‘o.d.e.’ approach for its stability and convergence, and implications of asynchrony. The learning schemes give stochastic approximation versions of the traditional iterative schemes for solving dynamic programs. Two learning schemes, Q-learning and the actor-critic method, are described along with their convergence analysis. As these ‘ideal’ schemes suffer from ‘curse of dimensionality’, one needs to use function approximation as a means to beat down the dimension to manageable levels. A function approximation based scheme is described for the simpler problem of policy evaluation. Some future research directions are pointed out.